Context¶
Buying and selling used phones and tablets used to be something that happened on a handful of online marketplace sites. But the used and refurbished device market has grown considerably over the past decade, and a new IDC (International Data Corporation) forecast predicts that the used phone market would be worth $52.7bn by 2023 with a compound annual growth rate (CAGR) of 13.6% from 2018 to 2023. This growth can be attributed to an uptick in demand for used phones and tablets that offer considerable savings compared with new models.
Refurbished and used devices continue to provide cost-effective alternatives to both consumers and businesses that are looking to save money when purchasing one. There are plenty of other benefits associated with the used device market. Used and refurbished devices can be sold with warranties and can also be insured with proof of purchase. Third-party vendors/platforms, such as Verizon, Amazon, etc., provide attractive offers to customers for refurbished devices. Maximizing the longevity of devices through second-hand trade also reduces their environmental impact and helps in recycling and reducing waste. The impact of the COVID-19 outbreak may further boost this segment as consumers cut back on discretionary spending and buy phones and tablets only for immediate needs.
Objective¶
The rising potential of this comparatively under-the-radar market fuels the need for an ML-based solution to develop a dynamic pricing strategy for used and refurbished devices. ReCell, a startup aiming to tap the potential in this market, has hired you as a data scientist. They want you to analyze the data provided and build a linear regression model to predict the price of a used phone/tablet and identify factors that significantly influence it.
Data Description¶
The data contains the different attributes of used/refurbished phones and tablets. The data was collected in the year 2021. The detailed data dictionary is given below.
Data Dictionary¶
brand_name: Name of manufacturing brand
os: OS on which the device runs
screen_size: Size of the screen in cm
4g: Whether 4G is available or not
5g: Whether 5G is available or not
main_camera_mp: Resolution of the rear camera in megapixels
selfie_camera_mp: Resolution of the front camera in megapixels
int_memory: Amount of internal memory (ROM) in GB
ram: Amount of RAM in GB
battery: Energy capacity of the device battery in mAh
weight: Weight of the device in grams
release_year: Year when the device model was released
days_used: Number of days the used/refurbished device has been used
normalized_new_price: Normalized price of a new device of the same model in euros
normalized_used_price: Normalized price of the used/refurbished device in euros
Import the necessary libraries¶
# Libraries to help with reading and manipulating data
import numpy as np
import pandas as pd
# Libraries to help with data visualization
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
# split the data into train and test
from sklearn.model_selection import train_test_split
# to build linear regression_model
import statsmodels.api as sm
# to check model performance
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
Import the dataset¶
used = pd.read_csv("used_device_data.csv")
Data overview¶
used.info()
We have 9 floating variables, 4 object variables, and 2 integer variables. Generally speaking, we can say that our data is composed by 11 numerical variables and 4 categorical ones.
used.describe(include="all").T
Exploratory Data Analysis¶
!pip install pandas-profiling==3.0.0 ##Installatio of the pandas profiling library.
from pandas_profiling import ProfileReport #Import the tool
report = ProfileReport(used, title = "Data exploration") #Report generation
report.to_notebook_iframe() #Deploy the report on the web
report.to_file("Report.html") #Generate a html file for the report
used.groupby("ram")["brand_name"].value_counts()
#create scatterplot of battery vs. brand
sns.displot(data=used,x="brand_name",y="battery",kind="hist",height=5,aspect=20)
used[used["screen_size"]>6.0].count()
sns.jointplot(data=used,x="brand_name",y="selfie_camera_mp")
Questions¶
Questions:
1.-What does the distribution of normalized used device prices look like?¶
It looks like a normal distributions with no skewing bias.
2.-What percentage of the used device market is dominated by Android devices?¶
An astonishing 93 % of the device market is dominated by android.
3.-The amount of RAM is important for the smooth functioning of a device. How does the amount of RAM vary with the brand?¶
Nokia tends to have a smaller RAM while Motorola and Huawei tend to have a larger RAM size.
4.-A large battery often increases a device's weight, making it feel uncomfortable in the hands. How does the weight vary for phones and tablets offering large batteries (more than 4500 mAh)?¶
In general, every brand has extreme values regarding the battery size, but the brands that have the larger battery sizes overall would be Apple, Google and Samsung.
5.-Bigger screens are desirable for entertainment purposes as they offer a better viewing experience. How many phones and tablets are available across different brands with a screen size larger than 6 inches?¶
About 3362 cell phones have screens wider tha 6 inches, making it the vast majority of the population.
6.-A lot of devices nowadays offer great selfie cameras, allowing us to capture our favorite moments with loved ones. What is the distribution of devices offering greater than 8MP selfie cameras across brands?¶
The distribution of selfie cameras accross brands seems to be skewed to the left heavily, so at least we can infere that it does not follow a normal distribution. The amount of cell phones with a greater than 8MP cameras is rather small in comparison to the ones with cameras of lesser qualities.
7.-Which attributes are highly correlated with the normalized price of a used device?¶
The normalized price of a new device, the screen sice, the selfie and main camera mp, and the battery.
Data preprocessing¶
used.isnull().any
There are missing data in some rows. Let's identify how many of them.
used.isnull().values.any()
used.isnull().sum()
Because the missing total number of values is less than 10 % of the observations, we can erase those values safely.
df = used.dropna(axis=0)
df.isnull().values.any()
df.describe().T
df.duplicated().sum
There are no duplicated values nor missing values. Now, we will do a new exploratory data analysis and check for outliers before building our model.
Exploratory Data Analysis II¶
reportII = ProfileReport(df, title = "Data exploration II") #Report generation
reportII.to_notebook_iframe() #Deploy the report on the web
reportII.to_file("ReportII.html") #Generate a html file for the report
As we can see in the correlation matrix, the variables that are most correlated with the price of a used device remain the same after quitting the missing and repeated values from our analysis. Meaning that removing those rows didn't affect our data. Now it's time to build up our model and train it to perfection.
Model Building: Linear Regression¶
x = df[["normalized_new_price","screen_size","battery","selfie_camera_mp","main_camera_mp","ram"]] #Entry variable
y = df["normalized_used_price"] #Variable de salida
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 0, shuffle = True )
print(x_test)
from sklearn.linear_model import LinearRegression #Librería para el modelo de regresión
model = LinearRegression()
model.fit(x_train,y_train)
columns_x = ["Normalized_new_price","main_camera_mp","ram","screen_size","battery","selfie_camera_mp"]
coefficients = pd.DataFrame(model.coef_, columns_x, columns=["Coeficientes"])
coefficients
y_predict = model.predict(x_test) #Predict the test values
Test = pd.DataFrame({"Real Value": y_test, "Prediction Value": y_predict, "Difference": y_test - y_predict})
Test
Data Preparation for Modeling¶
model = sm.OLS(y_train, x_train).fit()
print(model.summary())
Main insights from the model:¶
-The variable "Ram" is not statistically significant because its P value is less than its test value.
-There is risk of multicollinearity in th model according to the condition coefficient.
-With a kurtosis of 3 we can infere the model is normally distributed.
-The Durbin Watson has a higher score than the P value of 0.05. Meaning that there is no independence in the model.
-The Jarque Bera prob coefficient has a lower score than the P value of 0.05. Meaning that the model has a normal distribution.
# initial list of columns
predictors = x_train.copy()
cols = predictors.columns.tolist()
# setting an initial max p-value
max_p_value = 1
while len(cols) > 0:
# defining the train set
x_train_aux = predictors[cols]
# fitting the model
model = sm.OLS(y_train, x_train_aux).fit()
# getting the p-values and the maximum p-value
p_values = model.pvalues
max_p_value = max(p_values)
# name of the variable with maximum p-value
feature_with_p_max = p_values.idxmax()
if max_p_value > 0.05:
cols.remove(feature_with_p_max)
else:
break
selected_features = cols
print(selected_features)
x_train1 = x_train[selected_features]
x_test1 = x_test[selected_features]
mod2 = sm.OLS(y_train, x_train1).fit()
print(mod2.summary())
###Assumptions:¶
Although now all the variables are statistically significant, there are still some conditions we need to check and correct for our final model.
Multicollinearity
from statsmodels.stats.outliers_influence import variance_inflation_factor
X1 = df[["normalized_new_price","screen_size","battery","selfie_camera_mp","main_camera_mp"]]
vif_info = pd.DataFrame()
vif_info['VIF'] = [variance_inflation_factor(X.values, i) for i in range(X1.shape[1])]
vif_info['Column'] = X1.columns
vif_info.sort_values('VIF', ascending=False)
There are 4 variables with a VIF higher than 5, we need to drop them so that our model won't be biased with multicollinearity. Although, we can first drop the first three and check if the remaining two still have a high VIF value.
X2 = X1.drop(['screen_size', 'normalized_new_price', 'battery'], axis=1)
vif_info = pd.DataFrame()
vif_info['VIF'] = [variance_inflation_factor(X2.values, i) for i in range(X2.shape[1])]
vif_info['Column'] = X2.columns
vif_info.sort_values('VIF', ascending=False)
Now there are no variables that have a high VIF variable, up to now it is the most efficient model we can build.
X2
X2_train, X2_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 0, shuffle = True )
print(X2_test)
x3_test = x2_test[["selfie_camera_mp","main_camera_mp"]]
x2_train
x3_train = x2_train[["selfie_camera_mp","main_camera_mp"]]
model2 = LinearRegression()
model2 = sm.OLS(y_train, x3_train).fit()
print(model2.summary())
With the new model, there is no risk of multicolinearity, all coefficients are statistically significant and the R squared statistic increases. Meaning that the model now explains a higher percentage of the observations. The assumption of normality is safely assumed with a prob JB of 0 and a Kurtosis higher than the p value of 0.05. The Durbin Watson statistic is the last part of the model we want to check, to confirm that the model is homoscedastic.
Constant variance test¶
df_pred = x3_train[["selfie_camera_mp", "main_camera_mp"]]
df_pred["Actual Values"] = y_train # actual values
df_pred["Fitted Values"] = model2.fittedvalues # predicted values
df_pred["Residuals"] = model2.resid # residuals
df_pred.head()
sns.residplot(
data=df_pred, x="Fitted Values", y="Residuals", color="purple", lowess=True
)
plt.xlabel("Fitted Values")
plt.ylabel("Residuals")
plt.title("Fitted vs Residual plot")
plt.show()
There is no pattern in the plot of resifual variables, therfore we can assume linearity of the model.
import statsmodels.stats.api as sms
from statsmodels.compat import lzip
name = ["F statistic", "p-value"]
test = sms.het_goldfeldquandt(df_pred["Residuals"], x3_train)
lzip(name, test)
Since p-value > 0.05, we can say that the residuals are homoscedastic. So, this assumption is satisfied
Final Model¶
model2 = sm.OLS(y_train, x3_train).fit()
print(model2.summary())
In summary we can say that the two variables we analyzed are statistically significant and the model can explain 85 % of the actual values. Making it a relatively good model for predicting the price of used cell phones.
Additional insights and recommendations¶
We strongly reccomend the companies to concentrate developing better quality cell phones overall, but they should focus specieally on selfie and main cameras since those are apparently the most influential variables on the price of their cell phones.